A multimedia system is designed to present various multimedia materials in various combinations of text, graphics, video, image, animation, sound, etc. Such a system is a combination of hardware and software. The hardware includes a personal computer to which various multimedia devices can be attached. The hardware runs under the control of an operating system and multimedia application programs.
Multimedia applications impose heavy demands on the operating system to move large amounts of data from device to device, from system memory to a device, or vice-versa, in a continuous, real-time manner. Multimedia systems must support a flexible yet consistent means for transporting these large data objects, and control this activity accurately in real time. Adding new multimedia devices and data types should require minimal, if any, new system extension code. The total real memory requirements at run time must be minimized, so that system performance is not degraded. Also, support for complex data types and devices that manipulate interleaved data objects, must be provided. Finally, it must be possible to implement each multimedia data transport control means at the most appropriate system privilege level.
Operating system extensions that support multimedia data types and devices must provide the ability to control the many different multimedia I/O devices and to transport, or stream, large multimedia data objects within real-time constraints.
Multimedia applications control the input and output of large amounts of data, including the continual display of bitmaps, output of digitized audio waveforms or MIDI audio sequences, input of digitized audio from an analog microphone or line source, etc. The application controls all this data flow in the context of a real-time clock: certain events under program control must occur at explicitly defined points in time, and these times are defined very accurately (e.g., in milliseconds).
Given only the OS/2 control program services (DOS calls, or similar services in other operating systems such as AIX or UNIX), controlling this level of function at the application programming interface would require highly complex, device-specific, data transport control modules. Even if such modules were created, there would be no guarantee that the threads controlling each I/O operation would execute within its required time interval. To address this problem, the application would need to add sophisticated semaphore logic and make the I/O control threads time critical. The nature of multitasking operating systems, combined with the high data throughput load common to multimedia applications, will at times prevent data being delivered to the destination device within the allotted time interval. Failing to meet these real-time requirements will result in visible or audible defects in the multimedia presentation.
The real-time, continuous, high-volume data transport requirements present a set of interdependent problems that a generalized transport mechanism must solve:
1. Throughput-intensive Applications With Very Heavy Data I/O Demands
Uncompressed, digital motion video at 640.times.480.times.16 resolution requires approximately 3 Mb/sec. Even highly compressed, a digital motion video data stream can require transferring 60 Kb/sec. Adding 8 or 16-bit digital audio waveform data transfer into the scenario can increase the throughput load to 80 Kb/sec or greater. Supporting this throughput load continually while the multitasking system also performs other application functions such as file I/O and keyboard/mouse device control is the most important requirement for a generalized data streaming mechanism. But because the multimedia data presentations at the user interface (auditory and visual) are real-time dependant, these data transfers must occur within very tight real-time constraints.
2. Control of Multiple Different Hardware I/O Devices
Multimedia applications typically control a variety of I/O devices to manage presentations in the audio and video domains of the user interface. Audio devices that support waveform capture and playback, digital audio compact disc playback, Musical Instrument Digital Interface (MIDI) I/O, and voice-generation may be involved. Similarly, video devices that support NTSC (or PAL) video digitization, compression/decompression, and capture are routinely involved in multimedia presentation and/or authoring scenarios. The generalized data streaming mechanism must support these and future multimedia devices in a flexible manner, without requiring major modifications of the multimedia system extensions.
3. Provide Consistent Control Services Across Devices and Data Types
The data streaming mechanism must be controlled through services that are consistent across the different devices and data types involved in multimedia authoring and presentation scenarios. When new data types are introduced, as multimedia data standards evolve, this generalized streaming mechanism must not be substantially impacted. The control services must provide for easily incorporating new data types into existing applications. Likewise, these services must allow for seamlessly adding new I/O device types and capabilities, without impacting existing applications or requiring major revisions of the multimedia extensions.
4. Provide Identical Control Services at Multiple System Privilege Levels
Multimedia data objects may originate (or terminate) at a hardware I/O device, or in system (or application) memory. Where data streaming originates or terminates with a hardware device, the transport mechanism must be able to control behavior of the hardware device indirectly, rather than taking on direct control of the device, and thereby becoming device-specific itself. However, the nature of operating systems that support multiple privilege (or protection) levels--a common practice in modern operating systems--requires that for optimal performance the transport mechanism ("stream handler") should be attached to the device's native physical device driver at the highest system privilege level. This is the execution level where hardware device drivers typically run. Although it is desirable, for performance reasons, to create device driver level stream handlers, the stream handler device driver must not become hardware dependent itself. All hardware-specific code must be encapsulated in the device's native physical device driver. Alternatively, where no physical device is involved as the source or destination of the data stream, the stream handler may be implemented at a lower privilege level and still achieve optimal performance. Because the data transport mechanism is thus distributed across multiple privilege layers, identical stream control services must be provided at any level where the stream handlers execute. Multiple entry points at different privilege levels should be provided without incorporating redundant code.
5. Ability to Extend to New Devices/Data Types With Minimal Impact
New multimedia data standards and device types are being created almost continually. As this situation will persist for the indefinite future, the data transport mechanism must be able to easily and seamlessly adapt to support the new devices and data types.
6. Requirements for Physical Memory During Streaming Must be Kept to a Minimum
Despite the heavy throughput load imposed by multimedia applications, there should be conservation of memory resources at all times when data streaming is underway. Buffer management services should be provided so that applications do not need to become physical memory managers. However, since some applications create multimedia data objects in memory, these applications may need to have direct control over the memory buffers. This capability should be seamlessly supported by the data streaming mechanism and its control services.
7. CPU Loading Must be Kept to a Minimum During Streaming
Just as memory is a scarce resource that must be carefully managed, CPU cycles are another critical resource that cannot be squandered during multimedia application execution. The data streaming mechanism must keep CPU loads to a minimum by exploiting any direct hardware-to-hardware data transport capabilities (e.g., bus master data transfers). In addition, interrupt-driven device I/O should be exploited wherever possible to reduce CPU load, rather than allowing the data transport mechanism to control devices using device-polling techniques.
The following definitions are provided to establish a framework within which the present invention can be described:
Data Stream--A software means for data transfer through a data channel, in a continuous manner, from a source device (or memory address) to a target device (or memory address).
Dynamic Link Library (DLL)--In the OS/2 architecture, a 16-bit or 32-bit executable module that executes at privilege levels 2 or 3, and is loaded into memory in order to resolve the external references of another executable (e.g., .EXE) module. A DLL can execute only in a task context.
Physical Device Driver (PDD)--In the OS/2 architecture, a 16-bit executable module which executes at privilege level 0 (ring 0) and typically controls the operations of an I/O device. A PDD can execute in either a task context or a hardware interrupt context.
Stream--To continuously transfer data from a source to a destination (or target).